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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

Global  technological  advancements  necessitate  that  the  United  States  maintain  its 
commitment  to  technology  innovation  to  maintain  preeminence  in  national  security, 
global  projection  of  power  and  private  sector  innovation.  Secretaries  of  Defense  and 
industry  leaders  alike  have  shared  this  commitment.  Secretary  of  Defense  Ash  Carter  has 
made  a  commitment  “to  the  future — to  stay  ahead  of  a  changing  world,  to  stay 
competitive,  to  stay  aware  of  new  generations  and  attract  them  to  our  mission  of  serving 
the  country,  and  to  stay  abreast  of  technology”  (Carter  2015)  and  the  American  private 
sector  has  expressed  its  commitment  to  supporting  technology  development  by  outpacing 
federal  R&D  expenditures  beginning  in  the  late  1980s  (Figure  1). 

With  the  commitment  and  financial  resources  in  place,  the  execution  of  public 
and  private  sector  R&D  expenditures  requires  a  portfolio  selection  process  to  identify 
which  R&D  efforts  are  worthy  of  support.  While  there  are  numerous  R&D  portfolio 
selection  models  for  the  private  sector  (Henriksen  1999),  models  to  perform  the  same 
function  are  not  as  straightforward  for  the  public  sector.  Due  to  the  inherent  differences 
between  the  financial  structures  of  public  and  private  sector  organizations  (private  sector 
organizations  are  profit-driven,  whereas  public  sector  organizations  are  not),  this  thesis 
presents  an  R&D  portfolio  selection  process  and  optimization  model  for  use  by  public 
sector  entities  seeking  to  maximize  the  value  of  their  R&D  efforts  while  reducing  the  risk 
of  their  R&D  portfolios.  The  model  also  seeks  to  reduce  the  number  of  proposals  and 
resulting  time  required  for  individual  review.  The  resulting  model  includes  proposal 
filtering,  proposal  weighting,  proposal  valuation,  stochastic  proposal  risk  assessments  and 
optimization  all  designed  to  identify  the  best  use  of  a  public  sector’s  limited  resources. 
Finally,  the  model  is  applied  to  a  public  sector  DOD  case  study. 

B.  CURRENT  STATE  OF  R&D 

Federally  funded  research  and  development  (R&D)  combined  with  private  sector 
R&D  investments  result  in  technology  advancements  critical  to  the  national  economy  and 
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the  nation’s  defense.  In  particular,  federal  R&D  funding  has  been  a  critical  component  in 
the  overall  American  R&D  system  by  providing  the  necessary  funding  to  catalyze 
technology  development  in  areas  that  may  not  otherwise  receive  private  sector 
investment.  Federal  investment  enables  the  possibility  for  R&D  returns  to  far  exceed  the 
initial  investment.  For  example,  Larry  Page  and  Sergey  Brin  developed  the  algorithm  that 
ultimately  became  the  Google  search  engine  while  the  two  were  graduate  students  at 
Stanford  and  funded  with  a  part  of  a  $4.5  million  National  Science  Foundation  grant 
supporting  a  digital  library  effort  (Lane  2012).  The  revolutionary  Apple  iPod  leveraged 
lithium-ion  battery  technology,  liquid  crystal  displays  (LCDs),  digital  signal  processing 
and  magnetic  storage  devices,  all  of  which  were  developed  through  federally  funded 
basic  research  expenditures  (Lane  2012). 

Private  sector  R&D  funding  is  also  critical  to  an  organization’s  success  and 
competitiveness.  In  2006,  Microsoft  spent  $6.6  billion  on  R&D  with  Intel  and  IBM  each 
spending  approximately  $6  billion  on  R&D  efforts  (The  Economist  2007).  Methods  to 
place  a  value  on  these  investments  have  been  a  lingering  challenge,  particularly  for  R&D 
efforts  in  the  public  sector. 

1.  Domestic  R&D  Expenditures 

R&D  funding  in  the  United  States  for  both  private  and  public  sector  entities  is  a 
limited  resource  facing  perpetual  budgetary  pressure  and  contention  from  competing 
interests.  Private  sector  resources  invested  in  long  duration  R&D  efforts  reduce  near-term 
profits.  Therefore,  decreasing  R&D  spending  results  in  an  immediate  decrease  in 
expenses  and  an  increase  in  profits,  which  ultimately  satisfies  shareholders  and  corporate 
boards  (Dugan  2013).  Additionally,  basic  research  is  viewed  as  high  risk  and  its  output  is 
difficult  to  value  (Dugan  2013).  R&D  expenditures  by  the  federal  government,  while 
closely  aligned  with  private  sector  R&D  funding  from  1953  through  1988,  peaked  in 
2009  at  approximately  $127  billion  and  have  since  continued  to  decrease  through  2013 
(National  Science  Foundation  2016)  (Figure  1). 
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Figure  1.  U.S.  R&D  Expenditures,  by  Source  of  Funds:  1953-2013.  Source: 

National  Science  Foundation  (2016). 

In  1979,  private  sector  spending  surpassed  federal  R&D  spending  when  viewed  as 
a  total  percentage  of  R&D  expenditures  (Figure  2)  (National  Science  Foundation  2016). 
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Figure  2.  U.S.  total  R&D  Expenditures,  by  Source  of  Funds:  1953-2013.  Source: 

National  Science  Foundation  (2016). 
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For  general  comparison  purposes,  in  2015,  U.S.  federal  expenditures  equaled 
$3.8  trillion  (Figure  3).  The  three  major  expenses  amount  to  $938  billion  (25%)  on 
Medicare,  Medicaid  Children’s  Health  Insurance  Program  (CHIP)  and  the  Affordable 
Care  Act  (ACA)  marketplace  subsidies,  $888  billion  (24%  of  the  budget)  on  Social 
Security  payments,  and  $602  billion  (16%)  on  defense  and  international  security 
assistance  (United  States  Office  of  Management  and  Budget  2016)  while  in  2013  (the 
most  recent  figures  available),  the  federal  government  spent  approximately  $122  billion 
on  R&D. 
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Figure  3.  United  States  Federal  Budget  for  2015.  Source:  Center  on  Budget  and 

Policy  Priorities  (2016). 


2.  International  R&D  Expenditures 

The  current  state  of  international  R&D  funding  provides  insight  into  strategic 
efforts  by  allies  of  the  United  States,  as  well  as  near-peers  and  strategic  competitors. 
While  R&D  expenditures  by  the  United  States  lead  international  efforts,  China  is  rapidly 
increasing  its  R&D  expenditures  as  they  emerge  on  the  global  stage  (Figure  4).  The 
United  States’  investments  in  R&D,  as  a  percentage  of  its  gross  domestic  product  (GDP), 
remained  relatively  flat  between  1981  and  2013,  investing  between  2.25%  and  2.82%  of 
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GDP  on  R&D  ($457  billion  in  2013)  (National  Science  Foundation  2016).  For 
comparison,  China  invested  0.73%  of  its  GDP  on  R&D  in  1991  (the  first  year  these 
figures  were  available  for  that  country)  and  as  of  2013,  invested  2.08%  ($336  billion  in 
2013),  a  185%  increase  in  22  years  (National  Science  Foundation  2016). 


Gross  domestic  expenditures  on  R&D,  by  the  United  States,  the  EU, 
and  selected  countries:  1981-2013 
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Figure  4.  China’s  Rapid  Increase  in  Annual  R&D  Spending.  Source:  National 

Science  Foundation  (2016). 


In  response  to  the  changing  geopolitical  landscape  and  increased  R&D  efforts  by 

countries  like  China  and  Russia,  the  U.S.  Department  of  Defense  (DOD)  has  requested 

funding  for  a  “third  offset  strategy”  to  develop  capabilities  enabling  the  military  to 

counter  an  enemy  despite  rising  global  competition  (Gady  2016;  Eaglen  2016).  As 

Secretary  of  Defense  Chuck  Hagel  stated  in  his  offset  strategy  announcement,  “America 

does  not  believe  in  sending  our  troops  into  a  fair  fight”  (Hagel,  2014).  The  DOD  has 

sought  $71.8  billion  in  the  fiscal  year  2017  budget  to  conduct  R&D  with  $3.6  billion 

identified  for  the  third  offset  strategy.  This  strategy  has  identified  six  specific  investment 

areas  to  “identify  and  invest  in  innovative  ways  to  sustain  and  advance  America’s 
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military  dominance  for  the  21st  century.”  These  investment  areas  are:  access  and  area- 
denial,  guided  munitions,  undersea  warfare,  cyber  and  electronic  warfare,  human- 
machine  teaming,  and  war  gaming  and  development  of  new  operating  concepts  (Hagel 
2014).  With  billions  of  R&D  dollars  now  allocated  to  invest  in  an  R&D  portfolio  meeting 
the  DOD’s  strategic  objectives,  selecting  those  investments  in  an  effective  manner 
becomes  the  next  challenge. 

C.  THE  R&D  PORTFOLIO  SELECTION  PROBLEM 

The  R&D  portfolio  selection  problem  is  characterized  by  an  abundance  of  R&D 
proposals  of  varying  strategic  fit,  value  and  risk  all  seeking  limited  funding.  Since 
funding  entities  often  cannot  fund  all  the  submitted  proposals,  this  thesis  presents  criteria 
to  identify  and  select  proposals  meeting  an  organization’s  respective  strategic  goals. 

The  simplest  strategy  for  R&D  portfolio  selection  involves  analyzing  all 
proposals  applying  for  funding  in  a  given  year  by  evaluating  each  against  qualitative  and 
quantitative  criteria.  Proposals  are  then  prioritized  against  the  total  R&D  budget.  The 
highest  ranked  proposals  within  the  budget  are  selected,  and  those  falling  outside  the 
budget  are  either  tabled  until  the  following  year  or  denied.  This  strategy,  however,  fails  to 
factor  the  inherent  value  and  risk  associated  with  each  proposal. 

This  thesis  presents  an  alternative  strategy  involving  the  use  of  criteria  to  develop 
an  optimization  framework  aiding  proposal  analysis  and  selection  and  providing  a 
solution  to  the  R&D  portfolio  selection  problem.  An  optimized  R&D  portfolio  selection 
process  can  play  an  important  role  in  an  organization’s  strategic  success  by  enabling 
organizations  to  tailor  the  portfolio  to  align  with  their  strategic  needs.  Just  as  with 
financial  portfolios,  portfolio  analysis  and  optimization  enables  organizations  to  diversity 
their  investments  to  maximize  value  and  reduce  risk  across  the  portfolio.  Through 
optimization,  the  pool  of  R&D  proposals  can  be  analyzed  and  an  R&D  portfolio  can  be 
identified  and  selected,  meeting  a  funding  entity’ s  goals  while  maximizing  their  budget. 

There  are  several  R&D  portfolio  optimization  strategies  available  to 
organizations.  Most  strategies,  however,  are  better  suited  for  the  private  sector  due  to 
their  reliance  on  a  calculation  to  determine  a  proposal’s  value.  This  valuation  is  used  to 
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determine  if  the  proposal  should  be  included  in  the  R&D  portfolio  and  may  use  tools 
such  as  calculating  a  project’s  Net  Present  Value  (NPV),  Discounted  Cash  Flow  (DCF), 
Return  on  Investment  (ROI),  or  the  use  of  real  options  (Mun  2006). 

The  methods  available  to  public  and  private  sector  entities  for  R&D  proposal 
valuation  differ  due  to  the  nature  of  each  entity.  Private  sector  firms  are  measured  by 
their  financial  performance  and  position  within  highly  competitive  sectors,  so  their  R&D 
efforts  value  market  applicability  and  profit.  R&D  efforts  in  the  public  sector,  and  the 
DOD  specifically,  have  historically  valued  efforts  delivering  capabilities  either 
exclusively  intended  or  initially  intended  for  national  security  use  by  the  government 
such  as  stealth  technology  or  global  positioning  system  (GPS).  In  these  public  sector 
cases,  traditional  means  of  valuation  do  not  apply  due  to  the  lack  of  a  profit  concept.  In 
fact,  traditional  ROI  valuation  strategies  for  public  sector  R&D  efforts  were  abandoned  in 
the  1970s  and  1980s  by  NASA  and  the  National  Institute  of  Standards  and  Technologies 
(NIST)  due  to  “methodological  problems”  associated  with  efforts  to  calculate  the  outputs 
from  R&D  investment  (United  States  Government  Accountability  Office  1997). 
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II.  R&D  PORTFOLIO  SELECTION  MODEL 


A.  OVERVIEW 

The  research  of  this  thesis  culminates  in  the  creation  of  a  model  that  public  sector 
entities  can  apply  to  their  own  R&D  portfolio  optimization  problem  sets.  The  model  runs 
in  Microsoft  Excel,  and  incorporates  relevant  elements  of  Johnathan  Mun’s  Integrated 
Risk  Management  (IRM)  process.  A  visualization  of  the  process  is  provided  in  Figure  5. 
The  process  begins  with  a  funding  organization  creating  the  framework  to  establish  a 
competitive  R&D  funding  program.  Subsequently,  applicants  seeking  funding  to 
complete  R&D  efforts  must  submit  proposals  to  the  program,  creating  the  initial  portfolio 
of  proposals.  From  the  initial  portfolio,  the  proposals  are  filtered  down  and  weighted, 
reducing  and  shaping  the  portfolio  into  a  group  of  proposals  meeting  the  funding 
organization’s  mission  goals  and  strategic  objectives.  The  filtered  and  weighted  proposals 
are  each  assigned  a  value  and  risk  profile,  then  run  through  an  optimization  model  to 
identify  the  optimal  portfolio  makeup. 


Sponsoring 
organization 
(USN/USMC) 
generates  R&D  topics 


NPS 

Students/Faculty 
generate  proposals 
in  response  to  topics 


Initial  455  Topics 
and/or  Proposals 


NRP  solicits  SME-provided 
value  and  risk  assessments 
for  filtered  &  weighted 
dataset 


Filtered 

Proposals 


Filtered  & 
Weighted 
Proposals 


A 


§ 


V 


Solution  based  on 
given  objective 
function 


n  optimal 
proposals  within 
NRP  budget 


Figure  5.  Public  Sector  R&D  Portfolio  Selection  Process 
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The  model’s  goals  are  to  provide  public  sector  organizations  seeking  to  identify 
an  R&D  portfolio  with  the  means  to  make  the  most  of  their  resources  while  reducing  the 
risk  they  take  on.  The  primary  output  of  this  process  is  an  optimized  R&D  portfolio, 
meeting  an  organization’s  objective  function.  Secondary  outputs  include  a  means  to 
reduce  subjective  decisions  being  factored  into  the  portfolio,  and  a  reduction  in  the 
amount  of  time  spent  manually  reviewing  proposals  that  do  not  meet  an  organization’s 
criteria. 

B.  INTEGRATED  RISK  MANAGEMENT  PROCESS 

Mun’s  IRM  provides  a  comprehensive  and  sophisticated  framework  for  a  private 
sector  entity  to  develop  an  optimal  R&D  portfolio  and  serves  as  a  starting  point  for  the 
development  of  the  public  sector  R&D  portfolio  model.  Mun’s  IRM  includes  the 
following  eight  steps  (Mun  2006): 

1.  Qualitative  management  screening 

2.  Time-based  regression,  econometric,  and  stochastic  forecasting 

3.  Base  case  net  present  value  analysis 

4.  Monte  Carlo  simulation 

5.  Reed  options  problem  framing 

6.  Reed  options  modeling  and  analysis 

7.  Portfolio  and  resource  optimization 

8.  Reporting  and  update  analysis 

For  the  purposes  of  this  thesis,  five  of  the  eight  IRM  steps  were  used  due  to  their 
applicability  to  a  public  sector  financial  structure:  qualitative  management  screening, 
valuation,  risk  assessment,  portfolio  optimization  and  finally  reporting  and  update 
analysis.  The  forecasting  step  within  the  IRM  was  not  incorporated  in  the  model  since  it 
focuses  on  forecasting  future  revenues  and  related  sales  data  which  are  not  applicable  to  a 
public  sector  R&D  investment.  Additionally,  real  options  framing,  modeling  and  analysis 
were  omitted  from  the  model.  Real  options  identify  and  place  a  value  on  an 
organization’s  option  to  expand,  contract,  cease  or  switch  an  effort  and  require  the  use  of 
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a  discounted  cash  flow  model.  A  discounted  cash  flow  model  requires  cash  flow  to  be 
analyzed  and  forecasted,  which  is  a  financial  structure  a  public  sector  organization  also 
lacks. 


1.  Qualitative  Management  Screening 

The  IRM  framework  begins  by  conducting  “qualitative  management  screening”  to 
determine  the  strategic  fit  of  a  proposed  project  (Mun  2006).  An  organization  must 
determine  if  identified  proposals  align  with  the  organization’s  goals/mission  and  merit 
further  analysis.  An  outcome  of  this  screening  is  either  elimination  of  a  proposed  project, 
or  a  prioritization  assigned  by  a  sponsoring  organization’s  management.  To  help  facilitate 
objective  feedback  from  identified  experts  regarding  each  proposal,  the  Delphi  Method 
can  be  utilized.  The  Delphi  Method  strives  to  eliminate  groupthink  or  experts  consciously 
or  unconsciously  delivering  biased  input  to  a  group  seeking  to  make  a  collaborative 
decision.  By  replacing  the  group  setting  with  individual  questionnaires,  biases  are 
removed  (Helmer-Hirschberg  1967).  The  result  is  an  unbiased  assessment  of  the 
proposal.  If  utilized,  the  Delphi  Method  can  determine  which  of  the  proposed  projects 
should  be  included  in  the  process  leading  up  to  R&D  portfolio  optimization. 

The  public  sector  R&D  model  uses  qualitative  management  screening  in  the  form 
of  filtering  and  weighting  steps  to  identify  proposals  meeting  the  funding  organization’s 
strategic  objectives  or  criteria.  Proposals  that  do  not  meet  these  objectives  and  criteria  are 
removed  from  the  selection  process.  By  eliminating  proposals  lacking  the  strategic  fit  and 
goals  of  the  funding  organization,  the  resulting  portfolio  is  tailored  to  the  funding 
organization’s  goals  and  the  pool  of  proposals  requiring  subsequent  analysis  is  reduced. 
The  filtering  and  weighting  assessment  is  performed  in  accordance  with  specific 
established  criteria  defined  by  the  funding  program  and  verified  through  the  proposal 
application  process. 

The  information  solicited  by  the  funding  organization  as  part  of  the  application  is 
critical  to  ensure  the  model’s  utility.  First,  information  obtained  from  an  applicant  is 
filtered  to  eliminate  proposals  lacking  the  strategic  fit  or  specific  criteria  of  the 
organization.  Second,  the  information  submitted  by  an  applicant  has  weights  applied  to  it. 
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The  fields  an  applicant  must  fill  out  should  be  conducive  to  applying  filtering  and 
weighting  during  the  selection  process.  For  example,  requiring  applicants  to  provide  their 
anticipated  solution  timeframe  provides  program  representatives  objective  means  to  filter 
out  proposals  exceeding  the  sponsor’s  timeframe.  Additionally,  requiring  applicants  to 
identify  an  end  user  for  their  proposal  supports  technology  transfer  and  increases 
applicability  of  the  research.  This  identification  can  also  ensure  R&D  efforts  align  with 
the  sponsor’s  mission  and  provides  another  means  to  filter  out  a  proposal.  Other  filtering 
criteria  include  alignment  with  established  sponsoring  organization  objectives  such  as  the 
DOD’s  third  offset  strategy. 

The  weighting  process  is  appropriate  for  R&D  portfolios  lacking  interdependence 
among  the  projects  (Henriksen  1999).  The  criteria  for  weighting  is  funding  organization- 
dependent,  however,  good  criteria  for  weighting  purposes  involve  characteristics  which 
can  either  be  answered  yes/no  or  grouped  into  categories  such  as  low,  medium,  or  high. 
Weighting  criteria  should  also  be  independent  of  the  filtering  criteria  (although  the 
filtering  and  weighting  steps  can  be  consolidated  into  one  step — this  thesis  separates 
them  for  simplicity  of  explanation). 

Recommended  weighting  criteria  include  requiring  applicants  to  indicate  if  their 
proposal  is  supported  by  a  third-party  board  within  their  organization.  Another  weighting 
criterion  may  require  applicants  to  indicate  a  prioritization  of  their  efforts.  In  the  public 
sector  it  is  not  uncommon  for  the  entity  providing  the  R&D  funding  to  be  different  from 
the  entity  sponsoring  the  R&D  proposal  seeking  funding.  In  this  case,  the  sponsor  may 
determine  the  priority  that  can  then  be  weighted.  Other  weighting  criteria  include 
preferences  for  the  work  to  be  completed  by  an  interdisciplinary  team  composed  of 
operations,  logistics,  IT,  communications,  etc.,  or  a  team  of  a  certain  composition  such  as 
teams  with  new  hires  working  along  with  senior  mentors. 

Filtering  and  weighting  criteria  for  an  R&D  portfolio  selection  model  will  need  to 
be  evaluated  and  adapted  based  on  the  fields  a  funding  program  solicits  from  applicants. 
Once  the  criteria  are  complete,  however,  the  filtering  and  weighting  steps  can  effectively 
eliminate  proposals  falling  outside  an  R&D  program’s  target  efforts  and  the  portfolio  will 

begin  to  focus  on  those  proposals  aligning  with  the  program’s  strategic  goals. 
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2. 


Valuation 


The  following  two  steps,  valuation  and  risk  assessments,  are  completed  in  parallel 
and  are  applied  to  filtered  and  weighted  proposals.  An  approach  for  making  the  proposal 
selection  process  more  sophisticated  in  its  efforts  to  maximize  the  use  of  limited  funding 
involves  determining  a  valuation  for  filtered  and  weighted  proposals.  This  is  particularly 
difficult  for  the  public  sector.  The  private  sector  valuation  techniques  incorporate 
forecasted  revenues,  sale  prices,  sales  figures  and  production  quantities  to  develop  Net 
Present  Value  (NPV)  or  a  Return  on  Investment  (ROI)  for  a  project.  Since  the  public 
sector’s  financial  incentives  and  structure  differ  from  the  private  sector,  the  valuation  of  a 
proposal  in  the  R&D  portfolio  selection  model  leverages  subject  matter  expertise  and  is 
an  area  where  subjective  assessments  of  a  proposal  will  be  difficult  to  eliminate. 

Valuation  of  the  filtered  and  weighted  proposals  is,  for  the  purposes  of  the  model, 
defined  as  the  project’s  perceived  magnitude  of  success  based  on  a  subject  matter 
expert’s  assessment.  The  magnitude  of  success  of  each  proposal  is  broken  down  into  a 
five-category  Likert  scale:  very  low,  low,  medium,  high  and  very  high  with  a 
corresponding  figure  for  each  category;  very  low  equates  to  a  “1”  up  to  very  high 
equating  to  a  “5.”  This  magnitude  of  success  is  then  factored  into  subsequent  steps  of  the 
model. 


3.  Risk  Assessment 

Risk  is  viewed  in  terms  of  a  proposal’s  complexity  and  the  probability  of  a 
proposal  successfully  meeting  its  objectives.  The  Department  of  Defense’s  “Risk,  Issue, 
and  Opportunity  Management  Guide  for  Defense  Acquisition  Programs”  further  defines 
risk  by  identifying  “what  can  go  wrong”  and  the  impact  of  that  situation  (Department  of 
Defense  2015).  The  goal  for  the  model  is  to  reduce  the  risk  present  in  the  optimized 
model.  To  achieve  this,  each  of  the  filtered  and  weighted  proposals  are  assigned  a 
probability  of  success  value,  also  based  on  a  five  category  Likert  scale:  very  low,  low, 
medium,  high  and  very  high  and  again,  with  a  corresponding  figure  for  each  category; 
very  low  equates  to  a  “1,”  up  to  very  high  equates  to  a  “5.” 
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The  value  figure  of  each  proposal  is  then  multiplied  by  the  risk  figure  to  obtain  a 
proposal  “benefit.”  The  benefit  of  a  proposal,  however,  does  not  fully  reflect  the 
characteristics  of  a  proposal.  To  address  this,  the  benefit  of  each  proposal  is  divided  by 
the  cost  of  each  proposal  to  obtain  the  overall  cost-adjusted  benefit  ratio.  This  ratio  is 
then  factored  into  the  objective  function  of  the  next  step,  portfolio  optimization.  Table  1 
provides  an  example  of  this  process. 


Table  1.  Example  Risk  and  Valuation  Strategy  to  Obtain  the  Cost- Adjusted 

Benefit 


Applicant 
Provided  Data 

Value 

Risk 

Benefit 

Cost-Adjusted 

Benefit 

Magnitude  of  success 
(SME-provided) 

Probability  of  success 
(SME-provided) 

Proposal  Cost 

Magnitude 

Category 

Magnitude 

Figure 

Risk 

Category 

Risk  Figure 

Magnitude  x 
Risk 

Benefit  /  Cost 

1 

$60,000 

VL 

i 

L 

2 

2 

0.000033 

2 

$472,000 

VH 

5 

VH 

5 

25 

0.000053 

3 

$1,000,000 

H 

4 

M 

3 

12 

0.000012 

4 

$105,000 

M 

3 

H 

4 

12 

0.000114 

5 

$500,000 

H 

4 

L 

2 

8 

0.000016 

6 

$110,000 

VH 

5 

VH 

5 

25 

0.000227 

7 

$100,000 

H 

4 

VH 

5 

20 

0.000200 

4.  Portfolio  and  Resource  Optimization 

The  last  step  of  the  R&D  portfolio  selection  model  uses  the  cost-adjusted  benefit 
ratio  obtained  in  previous  step  as  input  for  the  objective  function  of  the  model. 
Specifically,  the  objective  function  is  to  maximize  the  sum  of  the  cost-adjusted  ratio 
based  on  binary  decisions  (whether  to  select  a  project  or  not)  subject  to  the  funding 
organization’s  budget. 
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5. 


Reporting  and  Update  Analysis 


Finally,  considering  the  complexity  of  the  model’s  framework  and  the  R&D 
portfolio  optimization  process,  deliberate  steps  to  effectively  communicate  the  results 
should  be  exercised.  AN  R&D  selection  exercise  by  BMW  highlights  the  importance  of 
identifying  a  “diffusing  agent”  to  communicate  the  details  of  the  process  for  increased 
acceptance  (Loch  2001).  In  addition,  for  continuity,  the  BMW  team  emphasizes 
educating  members  of  the  organization  as  a  form  of  technology  transfer  to  increase  the 
likelihood  of  the  R&D  portfolio  selection  method’s  perpetuity.  As  demonstrated  by 
BMW,  this  model  and  the  corresponding  complex  R&D  portfolio  optimization  process 
can  seem  like  a  black  box  to  those  unfamiliar  with  the  steps.  This  may  lead  to  dismissal 
of  the  results  by  key  stakeholders,  ultimately  defeating  the  purpose.  For  this  reason, 
deliberate  steps  to  effectively  communicate  the  results  of  the  model  should  be  exercised. 
Data  to  be  communicated  includes  a  profile  of  the  resulting  portfolio. 
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III.  NAVAL  POSTGRADUATE  SCHOOL’S  NAVAL  RESEARCH 

PROGRAM 


A.  INTRODUCTION 

Recognizing  the  value  of  continued  R&D  funding  for  U.S.  Navy  and  Marine 
Corps  strategic  positioning,  the  Secretary  of  the  Navy  established  the  Naval  Postgraduate 
School’s  Naval  Research  Program  (NRP)  in  2013  to  research  new  capabilities  and  topics 
on  behalf  of  the  Navy  and  Marine  Corps.  The  program  leverages  the  knowledge  and 
expertise  found  at  the  Naval  Postgraduate  School  to  solve  problems  related  to  naval 
research  and  “supports  the  Navy  in  reaching  well-informed,  objective  decisions  on 
strategic,  operational,  and  programmatic  issues  through  collaborative  research,  which 
integrates  academic  studies  and  analysis  with  advanced  decision-support  tools”  (Naval 
Postgraduate  School  2013). 

The  program  has  proven  very  popular,  attracting  527  proposals  in  FY  16  and  421 
proposals  for  FY  17  (Figure  6).  The  NRP  faces  the  fundamental  R&D  portfolio  selection 
problem;  despite  the  popularity  and  abundance  of  proposed  solutions,  not  all  the 
proposals  can  be  funded.  The  NRP’s  FY  17  annual  budget  is  $13,261,000,  while 
submissions  for  all  FY  17  topics  totaled  $30,826,111.  As  a  result,  the  NRP  R&D 
portfolio  selection  process  is  an  excellent  case  study  and  opportunity  to  develop  and 
implement  an  optimization  model  tailored  to  the  public  sector. 


FY  17  Submission  Estimates 

$500,000 


Figure  6.  Requested  Funding  Profile  for  all  421  FY  17  Submissions  (note:  gaps 
along  horizontal  axis  reflect  topics  lacking  budgets) 


17 


B.  NRP  PROCESS 


Most  Naval  Research  Program  topics  are  research  questions  developed  by  sponsor 
organizations  from  the  U.S.  Navy  or  U.S.  Marines.  Additionally,  NPS  faculty  are  able  to 
propose  their  own  research  question,  but  these  must  be  adopted  by  a  sponsor  to  be 
funded.  After  the  research  questions  are  submitted,  NPS  faculty  then  develop  proposals 
for  posted  topics  and  generate  budget  estimates  to  solve  the  research  problem. 
Occasionally,  multiple  NPS  faculty  provide  proposals  for  the  same  problem.  When  this 
occurs,  each  proposal’s  budget  may  differ  depending  on  the  applicant’s  proposed 
solution.  In  some  instances,  sponsor-provided  topics  remain  unaddressed  by  NPS  faculty, 
resulting  in  the  absence  of  a  proposed  solution  and  budget.  For  example,  while  there  were 
421  proposals  for  the  FY  17  NRP,  only  255  possessed  a  proposed  solution  and  associated 
budget,  and  were  removed  from  the  portfolio  as  part  of  the  effort  to  identify  an  optimal 
portfolio. 

The  NRP  requires  applicants  to  provide  traditional  administrative  data  about  their 
proposals,  but  due  to  the  inability  to  fund  all  the  proposals,  applicants  are  also  asked  to 
provide  data  that  aids  in  the  filtering  and  weighting  of  the  proposals.  Administrative 
fields  include  a  topic  title,  topic  description,  multiple  potential  research  focus  areas  and 
whether  the  topic  is  well  suited  for  a  thesis  or  a  broad  area  study.  Applicants  are  also 
required  to  inform  the  NRP  administrators  if  their  proposal  has  received  advance  high- 
level  support  from  the  Navy’s  Resources  and  Requirements  Review  Board  (R3B),  staffed 
by  Navy  Admirals,  or  the  Marine  Requirements  Oversight  Council  (MROC),  staffed  by 
Marine  Corps  Generals.  The  NRP  acknowledges  that  confirmation  of  review  by  either  of 
these  boards  will  place  a  greater  priority  on  a  proposal.  Given  the  high  level  of  each  of 
these  boards,  only  twenty  submissions  had  obtained  the  support  of  these  boards  and  of 
these,  only  eleven  were  associated  with  proposals  that  had  budgets  ($1,125,000  for  all 
eleven).  The  total  of  $1,125,000  is  well  within  the  NRP’s  budget,  so  additional  evaluation 
criteria  is  required  to  determine  which  other  non-R3B  and  MROC  supported  proposals 
should  be  selected. 

A  second  criteria  used  to  evaluate  the  proposals  is  whether  or  not  they  meet  stated 

Navy  or  Marine  S&T  objectives  as  determined  by  the  Office  of  Naval  Research  (ONR). 
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Of  the  255  proposals  with  budgets,  116  meet  the  ONR’s  S&T  objectives.  The  ONR 
topics  align  with  the  DOD’s  third  offset  strategy  and  proposals  contributing  to  one  of 
these  areas  will  receive  a  higher  weighting: 

1.  Assure  Access  to  Maritime  Battle  space 

2.  Autonomy  and  Unmanned  Systems 

3.  Electromagnetic  Maneuver  Warfare 

4.  Expeditionary  and  Irregular  Warfare 

5.  Information  Dominance  ( Cyber) 

6.  Platform  Design  and  Survivability 

7.  Power  and  Energy 

8.  Power  Projection  and  Integrated  Defense 

9.  Warfighter  Performance 

A  third  criteria  used  to  evaluate  the  proposals  in  the  NRP  dataset  is  a  prioritization 
of  the  topic  as  determined  by  the  sponsor  organization.  The  sponsoring  organization 
classifies  topics  as  either  “Low,”  “Medium”  or  “High.”  This  prioritization,  however,  does 
not  provide  a  substantial  filtering  capability  because,  as  expected,  the  majority  of  the 
proposals  have  been  deemed  “High”  priority  by  their  sponsoring  organization;  196 
received  a  “High”  prioritization  out  of  the  255  proposals  with  budgets. 

Viewed  across  all  three  criteria,  the  twenty  overall  R3B/MROC  supported 
submissions  are  reduced  to  fourteen  when  you  apply  the  constraint  that  they  must  be  meet 
Navy/Marine  S&T  objectives  and  they  be  deemed  a  “High”  priority  by  the  topic’s 
sponsor.  Of  these  elite  fourteen  submissions,  only  eight  were  associated  with  a  solution 
resulting  in  a  proposal  budget  and  the  opportunity  to  be  funded. 

With  a  budget  of  $13.261M  to  fund  the  FY  2017  proposals,  the  NRP  can  simply 
winnow  down  the  list  of  proposals  from  255  at  a  total  portfolio  cost  of  $30,826,111  by 
applying  criteria  filters  until  the  total  number  of  proposals  is  maximized  within  the 
budget  constraint.  This  strategy,  however,  does  not  account  for  the  value  or  risk  each 
proposal  poses  to  the  NRP.  Similarly,  each  proposal  represents  a  different  risk  to  the 
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NRP.  Since  the  NRP  is  structured  as  a  grant- arrangement,  if  a  proposal  exceeds  its 
budget,  additional  funds  are  not  available  to  an  applicant  to  complete  the  proposed  work. 
If  a  funded  proposal  exceeds  its  budget,  goals  set  forth  in  the  proposal  are  not 
accomplished.  As  a  result,  the  NRP  seeks  to  make  the  best  use  of  its  funding,  by 
maximizing  the  value  derived  from  the  pool  of  selected  proposals,  while  minimizing  the 
risk  present  in  each  of  the  selected  proposals  not  meeting  its  objectives. 
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IV.  MODEL  IMPLEMENTATION  WITH  NRP  DATA  SET 


The  NRP  dataset  served  as  a  case  study  for  the  implementation  of  the  R&D 
portfolio  selection  process  developed  by  this  thesis.  To  begin,  a  copy  of  the  FY  17  NRP 
dataset  was  obtained  in  the  form  of  a  Microsoft  Excel  file.  The  same  filtering,  weighting, 
valuation,  risk  and  optimization  steps  were  applied  with  the  objective  of  maximizing  the 
value  of  selected  proposals  while  minimizing  the  risk  present.  As  an  added  benefit,  the 
filtering  and  weighting  steps  minimize  both  the  subjective  input  applied  during  the 
proposal  selection  processes  and  reduce  the  amount  of  time  NRP  proposal  evaluators  are 
required  to  spend  on  proposal  evaluation.  The  following  steps  describe  the  process 
followed  to  analyze  the  NRP  dataset  and  develop  the  optimization  model  as  depicted  in 
Figure  7. 


Sponsoring 
organization 
(USN/USMC) 
generates  R&D  topics 


NPS 

Students/Faculty 
generate  proposals 
in  response  to  topics 


SME-provided  value  and  risk 


Solution  based  on 
given  objective 
function 


n  optimal 
proposals  within 
sponsor 
organization 
budget 


Figure  7.  R&D  Portfolio  Selection  Process  for  Public  Sector  Entities  Applied  to 

NRP  Dataset 
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A.  FILTERING  PROCESS  USING  QUALITATIVE  MANAGEMENT 
SCREENING 

An  additive  filtering  process  was  utilized  to  implement  the  qualitative 
management  screening  step  from  the  Integrated  Risk  Management  framework  and 
analyze  the  dataset  across  various  filtering  criteria.  The  NRP  dataset  was  incorporated 
into  a  Microsoft  Excel  spreadsheet  and  analyzed  by  incorporating  pivot  tables  into  the 
Excel  workbook  for  efficient  evaluation.  The  first  filter  applied  determined  whether  or 
not  a  budget  had  been  generated  for  the  respective  topic.  If  NPS  faculty  had  not  identified 
a  solution  for  a  given  topic,  a  budget  was  not  generated,  and  the  proposal  was  not 
incorporated  into  the  filtering  or  subsequent  weighting  process.  The  application  of  this 
first  filter  resulted  in  the  total  number  of  proposals  being  reduced  from  421  to  255  (the 
total  portfolio  cost,  however,  remained  the  same  because  this  filter  simply  removed 
budgets  of  zero). 

Various  filtering  combinations  were  evaluated  to  determine  the  corresponding 
portfolio  count  and  total  portfolio  cost  (Table  2  and  Figure  8).  While  the  additive  use  of 
filtering  can  be  a  relatively  simple  evaluation  method  for  final  portfolio  selection,  it 
assumes  an  equal  weight,  value  and  risk  have  been  applied  to  all  of  the  eligible  proposals. 
After  analyzing  multiple  combinations,  filter  method  #4  (Table  2)  was  selected, 
eliminating  proposals  lacking  a  budget  or  prioritized  “Low”  or  “Medium”  by  their 
sponsoring  organization.  This  reduces  the  portfolio  down  from  421  proposals,  to  196  with 
a  total  cost  of  $24,315,927.  Additional  filters  were  not  implemented  because  there  were 
no  subsequent  filtering  criteria  that  if  applied,  kept  the  overall  portfolio  above  the  NRP’s 
overall  budget,  therefore  defeating  the  purpose  of  optimizing  a  portfolio  for  a  given 
objective  function.  Proposals  remaining  in  the  dataset  after  filtering,  are  subject  to  the 
weighting  step,  which  enables  another  evaluation  perspective. 
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Table  2.  2017  NRP  Proposal  Breakdown  by  Increasing  Filtering  Constraint 

Reflecting  Resulting  Proposal  Count  and  Portfolio  Cost 


Filter 

Method 

Filter  Description 

Proposal 

Count 

Portfolio  Cost 

1 

No  filtering 

421 

$30,826,111.00 

2 

Only  those  submissions  with  a  budget 

255 

$30,826,111.00 

3 

Only  those  submissions  with  a  budget  and 
prioritized  as  “Medium”  or  “High”  by  their 
sponsor  organization 

246 

$30,000,111.00 

4 

Only  those  submissions  with  associated 
budgets  and  prioritized  solely  as  “High”  by 
their  sponsor  organization 

196 

$24,315,927.00 

5 

Only  those  submissions  with  associated 
budgets  and  prioritized  as  “High”  by  their 
sponsor  organization  and  meeting  Navy/ 

Marine  S&T  objectives 

90 

$10,806,600.00 

6 

Only  those  submissions  with  associated 
budgets  and  prioritized  as  “High”  by  their 
sponsor  organization  and  meeting  Navy/ 

Marine  S&T  objectives  and  confirming  R3B/ 
MROC  supported  topics 

8 

$765,000.00 

7 

Only  R3B/MROC  supported  submissions  with 
a  budget 

11 

$1,125,000.00 

ro 
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Figure  8.  2017  NRP  Proposal  Breakdown  by  Increasing  Filtering  Constraint  with 

Resulting  Proposal  Count  and  Portfolio  Amount 
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B.  WEIGHTING  PROCESS 


The  weighting  process  (Table  3)  leverages  the  analysis  completed  in  the  filtering 
step  and  supplements  the  results  with  a  weighting  criteria  to  complete  further  objective 
selection  within  the  NRP  dataset.  If  a  proposal  was  supported  by  the  R3B  or  MROC,  it 
received  a  weight  of  “5”  to  force  these  proposals  to  stand  out  from  non-R3B  or  MROC 
supported  proposals  due  to  the  high  profile  support.  Topics  not  supported  by  the  R3B  or 
MROC  received  a  weight  of  “0.”  Proposals  meeting  Navy/Marine  S&T  objectives 
received  a  weight  of  “3”  and  a  “0”  otherwise.  The  Topic  Sponsor  Organization  Priority 
category  served  as  a  weighting  category,  as  well  as  a  filtering  criteria  because  in  some 
instances,  despite  a  proposal  being  prioritized  as  “High,”  it  was  not  R3B/MROC 
supported,  and  also  lacked  alignment  with  Navy/Marine  S&T  objectives.  Without 
retaining  Topic  Sponsor  Prioritization  as  a  weighting  criteria,  these  proposals  would  have 
no  weight  otherwise. 


Table  3.  NRP  Dataset  Weighting  Example 


Topic  ID 

R3B  or  MROC 
Supported 

Navy/Marine  S&T 
Objectives 

Topic  Sponsor 
Priority 

R3B  or 

MROC 

Supported? 

Navy/Marine 

S&T 

Objectives? 

Topic 

Sponsor 

Organization 

Priority 

Weighted 

Score 

NPS-17-M052 

High 

- 

- 

3 

3 

NPS-17-M057 

Yes 

High 

- 

3 

3 

6 

NPS-17-M071 

Yes 

Yes 

High 

5 

3 

3 

11 

NPS-17-M078 

Yes 

High 

- 

3 

3 

6 

NPS-17-M087 

High 

- 

- 

3 

3 

The  weights  applied  for  each  proposal’s  R3B  or  MROC  Support,  Navy/Marine 
S&T  Objectives  and  Topic  Sponsor  Priority  were  then  summed  to  determine  the  topic’s 
overall  weighted  score.  Resulting  scores,  as  depicted  in  Figure  9,  range  from  “3”  where 
the  only  weight  applied  was  due  to  a  proposal  being  ranked  “High”  by  their  sponsoring 
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organization,  to  “11”  for  proposals  with  R3B  or  MROC  support,  which  met  Navy/Marine 
S&T  objectives  and  were  prioritized  as  “High”  by  their  respective  topic  sponsor 
organization. 


Portfolio  Weighting  Distribution 
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Figure  9.  FY  17  NRP  Proposal  Weights  (196  weighted  proposals  post-filtering) 


Additionally,  this  objective  weighting  process  reduced  the  NRP  portfolio  from 
255  proposals  requiring  detailed  review,  down  to  a  portfolio  consisting  of  196  proposals. 
The  reduction  of  the  number  of  proposals  from  255  down  to  196  also  represents  a 
considerable  time  savings  for  subsequent  evaluation.  Assuming  20  minutes  per  proposal 
review,  this  results  in  a  time  savings  of  nearly  20  hours  to  review  the  weighted  portfolio. 
Considering  that  this  review  is  likely  to  be  completed  by  more  than  one  person,  this  time 
savings  can  be  multiplied  by  each  person  reviewing  the  portfolio,  further  increasing  the 
benefits  of  reducing  the  portfolio  early  in  the  selection  process.  Members  of  the  NRP  are 
not  assigned  to  the  NRP  full  time,  so  any  time  savings  they  can  realize  in  their  efforts  to 
support  the  NRP  selection  process  enables  them  to  resume  their  primary  professional 
roles.  In  addition,  by  reducing  the  amount  of  time  each  reviewer  is  required  to  support  the 
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review  process,  the  NRP  would  likely  be  able  to  solicit  the  input  of  additional  reviewers, 
thus  providing  a  more  diverse  and  comprehensive  perspective  on  proposal  evaluation. 


C.  PROPOSAL  VALUE 

Placing  a  value  on  a  public  sector  investment  cannot  be  approached  in  the  same 
manner  as  valuing  an  investment  in  the  private  sector.  ROI,  NPV,  DCF  are  not 
appropriate  means  to  value  a  public  sector  R&D  project  due  to  the  lack  of  appropriate 
financial  metrics.  Since,  for  example,  ROI  measures  an  investment’s  profitability,  it  is  not 
appropriate  for  the  public  sector  since  profitability  is  not  part  of  the  public  sector’s 
financial  structure.  Net  present  value  factors  a  discount  rate,  or  the  targeted  interest  rate 
an  entity  needs  to  make  on  an  amount  of  money  today,  to  reach  a  targeted  amount  of 
money  in  the  future,  into  a  calculation  to  determine  if  an  investment  is  worth  pursuing; 
discount  rates  and  interest  rates  are  not  part  of  a  public  sector  entity’s  financial  structure. 
Finally,  discounted  cash  flow  is  used  to  estimate  an  investor’s  anticipated  revenues,  based 
on  the  time  value  of  money  (assumes  a  dollar  today  is  worth  more  than  a  dollar 
tomorrow),  factoring  in  interest  rates — again,  a  metric  the  public  sector  lacks. 

As  a  result,  and  for  the  purposes  of  the  case  study  to  demonstrate  the  model 
process,  valuations  for  the  proposals  were  simulated  to  replicate  an  assessment  by  a 
subject  matter  expert  (SME).  These  valuations  were  applied  across  a  five-point  Likert 
scale  of  very  low,  to  very  high,  with  a  number,  one  through  five,  assigned  to  each. 
Subject  Matter  Experts  could  use  various  criteria  to  determine  which  value  to  apply. 
These  could  include  impact  to  the  sponsoring  organization,  improvement  over  current 
capabilities  or  cross-service  benefit.  These  associated  value  figures  were  then  used  to 
obtain  a  benefit  figure  in  collaboration  with  the  outcome  of  the  subsequent  risk  step. 

D.  RISK  EVALUATION  OF  PROPOSALS 

Each  of  the  filtered  and  weighted  proposals  were  then  evaluated  for  their 

probability  of  success.  As  with  the  valuation  step,  this  assessment  would  be  completed  by 

SMEs,  however  for  the  purposes  of  this  case  study,  the  risk  assessments  (also  completed 

on  a  five  point  Likert  scale  from  very  low  probability  of  success  to  very  high  probability 

of  success)  were  simulated.  As  with  the  valuation  step,  proposals  assessed  to  be  of  very 
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low  probability  of  success  were  assigned  a  figure  of  “1”  up  to  “5”  for  proposals  assessed 
to  be  of  very  high  probability  of  success. 

Upon  completion  of  the  risk  and  value  assessments,  the  resulting  value  figure  and 
resulting  risk  figure  are  multiplied  by  each  other  to  determine  an  overall  benefit.  The 
benefit  figure  for  each  proposal  is  then  divided  by  the  cost  of  the  respective  proposal  to 
calculate  a  cost-adjusted  benefit  ratio,  which  is  then  used  in  the  optimization  step  (Table 

4). 

E.  OPTIMIZATION 

The  final  step  leveraged  the  results  of  all  prior  steps  to  identify  an  optimal  R&D 
portfolio  for  the  NRP.  Several  optimization  strategies  were  evaluated,  all  subject  to 
remaining  within  the  NRP’s  annual  budget.  These  strategies  included  maximizing  the 
optimized  portfolio  cost,  maximizing  the  optimized  portfolio  value  figure,  minimizing  the 
portfolio’s  risk  figure  and  maximizing  the  portfolio’s  percent  obligated.  These 
approaches,  however,  did  not  address  the  benefits  of  consolidating  multiple  criteria  by 
selecting  a  portfolio  representing  the  greatest  cost-adjusted  benefit. 

The  resulting  objective  function  incorporates  the  valuation  and  risk  assessments 
completed  in  previous  steps  to  maximize  the  sum  of  the  individual  cost-adjusted  benefit 
ratios  for  all  filtered  and  weighted  proposals,  subject  to  the  annual  NRP  budget.  Decision 
variables  for  optimization  were  defined  as  binary  where  if  a  proposal  was  selected  for  the 
final  portfolio,  the  solution  reflected  a  “1”  and  a  “0”  otherwise.  An  example  of  the 
optimization  process  is  provided  in  Table  4. 
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Table  4.  Example  Optimization  Process;  Maximize  Sum  of  Cost-Adjusted  Benefit  Ratio  Subject  to  $500,000  Maximum 

Investment 


UiartArea  | 

Applicant  Provided  Data 

Value 

Risk 

Benefit 

Cost- 

Adjusted 

Benefit 

Optimization 

Magnitude  of 
success  (SME- 
provided) 

Probability  of 
success  (SME- 
provided) 

Project 

Selected 
(1  =  yes, 

0  =  no) 

Selected 

Project  Cost 

Selected  Project  Cost- 
Adjusted  Benefit  Ratio 

Proposal  Number 

Proposal  Cost 

Magnitude 

Category 

Resulting 

Figure 

Risk 

Category 

Resulting 

Figure 

Benefit/Cost 

NPS-17-M001 

$60,000 

VL 

i 

L 

2 

2 

0.000033 

1 

$60,000 

0.000033 

NPS-17-M005 

$50,000 

VH 

5 

VH 

5 

25 

0.000500 

1 

$50,000 

0.000500 

NPS-17-M014 

$220,000 

H 

4 

M 

3 

12 

0.000055 

0 

$- 

0 

NPS-17-M022-A 

$175,000 

M 

3 

H 

4 

12 

0.000069 

0 

$- 

0 

NPS-17-M022-B 

$95,000 

H 

4 

L 

2 

8 

0.000084 

1 

$95,000 

0.000084 

NPS-17-M051 

$100,000 

VH 

5 

VH 

5 

25 

0.000250 

1 

$100,000 

0.000250 

NPS-17-M052 

$95,000 

H 

4 

VH 

5 

20 

0.000211 

1 

$95,000 

0.000211 

Total 

$795,000 

$400,000 

Obj  Fxn:  Max  Sum  of 
Cost-Adjusted  Benefit 
Ratio 

Subject  to: 

<= 

Max  Program  Budget: 

$500,000 

0.0010781 
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F.  ANALYSIS 

The  optimization  process  resulted  in  an  interesting,  but  not  an  unexpected 
outcome.  Considering  the  objective  function,  proposals  with  higher  value  and  higher 
probabilities  of  success  (lower  risk)  were  selected.  Figure  10  reflects  the  proposal 
valuation  breakdown  by  selected  versus  non-selected  proposals.  Figure  11  reflects  the 
proposal  probability  of  success  breakdown  by  selected  versus  non-selected  proposals. 
Finally,  the  portfolio’s  cost-adjusted  benefit  ratios  are  depicted  in  Figure  12,  with  a 
Paredo  frontier  appearing  around  0.00005. 


Figure  10.  Proposal  Valuation  Distribution 
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Cost-Adjusted  Benefit  Proposal  Count 


Portfolio  Probability  of  Success  Distribution 


Figure  1 1 .  Proposal  Probability  of  Success  Distribution 


Portfolio  Results 


•  Proposal  Not  Selected  •  Proposal  Selected 


Figure  12.  Proposal  Distribution  across  Cost-Adjusted  Benefit  Ratio 
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G. 


RECOMMENDATIONS 


To  better  assess  a  proposal’s  probability  of  failing  to  meet  its  stated  objectives, 
and  refine  the  risk  assessments  for  each  proposal,  future  NRP  funding  applications  may 
wish  to  specifically  require  an  applicant  to  submit  what  the  anticipated  deliverables  will 
be  should  their  proposal  be  selected  for  funding.  Proposals  focusing  on  completing  a 
study  or  developing  an  analytical  product  would  likely  specify  their  intent  to  deliver  a 
written  report  along  with  the  delivery  of  accompanying  software  models  (Excel,  Risk 
Simulator,  MATLAB,  etc.).  Alternatively,  proposals  identifying  a  physical  technological 
product  to  be  built  could  identify  the  complexity  of  the  proposed  system  or  Technical 
Readiness  Levels  (TRLs)  of  the  system,  its  assemblies  or  components.  Highlighting  a 
TRL  would  identify  the  complexity  of  a  proposed  development  or  integration  effort, 
better  informing  the  risk  assessment.  For  both  technical  developments  and  research- 
oriented  proposals,  risk  assessments  could  include  applicant-provided  risk  identification 
and  mitigation  plans.  Research-oriented  topics  could,  for  example,  identify  the 
anticipated  source  of  data  to  be  analyzed  and  an  assessment  of  the  difficulty  in  obtaining 
that  dataset.  This  would  also  aid  NRP  proposal  evaluators  in  making  risk  assessments 
considering  that  one  of  the  biggest  problems  in  data  analysis  is  obtaining  the  data  in  the 
first  place.  Similarly,  technical  developments  may  identify  the  risk  associated  with 
integrating  a  commercial  piece  of  hardware  into  a  DOD  system.  Having  this  additional 
data  would  likely  aid  the  NRP  staff  and  SMEs  assigned  as  they  evaluate  the  inherent  risk 
of  the  project  being  successfully  completed. 

Additionally,  the  existing  NRP  application  contains  fields  that  could  be  leveraged 
as  weighting  criteria  if  the  NRP  chose  to  expand  the  model’s  weighting  criteria.  The 
inclusion  of  additional  metrics  to  improve  the  weighting  capability  of  the  overall 
portfolio  optimization  process  could  further  reduce  the  size  of  the  portfolio  requiring 
subsequent  NRP  manual  review.  For  example,  if  the  NRP  were  able  to  identify  which 
NPS  academic  departments  valued  Broad  Area  Studies  over  Thesis  Topics,  and 
correlated  each  proposal  to  the  departments,  a  weighting  structure  could  be  applied  to 
these  criteria,  based  on  the  department’s  perspective  of  the  topic  type.  Another 
recommended  weighting  option  is  the  inclusion  of  criteria  for  interdisciplinary  or  team 
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composition  to  provide  additional  factors  for  weighting  a  proposal  and  would  likely  aid 
in  differentiating  the  initial  pool  of  proposals. 
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V.  CONCLUSION 


Maintaining  our  nation’s  technologic  edge  is  a  persistent  requirement  and  has 
historically  been  led  by  or  supported  by  public  sector  investment.  The  Department  of 
Defense  has  committed  to  financially  support  this  edge  by  allocating  billions  of  dollars 
towards  its  “third  offset  strategy.”  With  such  large  sums  of  money  and  the  strategic 
advantages  provided  by  the  technology  developed  by  this  funding  at  stake,  efficient 
strategies  must  be  used  to  make  the  best  use  of  limited  resources. 

This  thesis  has  presented  one  such  method,  and  applied  it  to  an  R&D  effort 
sponsored  by  the  Naval  Research  Program  (NRP)  as  a  case  study.  The  method  reinforces 
objective  means  of  proposal  selection,  while  integrating  subjective  feedback  from  experts 
to  support  the  proposal  identification  steps  leading  to  an  optimal  R&D  portfolio  being 
selected.  The  methods  included  filtering  and  weighting  steps,  assessing  a  proposal’s 
value,  assessing  a  proposal’s  risk  and  culminated  in  an  optimization  effort  identifying  an 
R&D  investment  portfolio  meeting  a  specific  objective  function.  The  objective  function 
sought  to  maximize  the  sum  of  the  chosen  proposals’  cost-adjusted  benefit  calculation 
subject  to  the  NRP’s  annual  budget.  The  results  of  the  optimization  identified  133 
proposals.  The  results,  as  expected,  predominately  chose  higher  valued  proposals 
characterized  as  lower  risk  (higher  probability  of  success). 

The  key  takeaway  from  this  overall  effort  is  that  while  R&D  portfolio  selection 
can  leverage  many  of  the  private  sector’s  techniques  to  identify  an  optimal  R&D 
investment  portfolio,  the  valuation  strategies  are  unique  and  not  interchangeable.  This 
difference,  however,  is  not  insurmountable,  and  valuation  and  risk  efforts  for  the  public 
sector  can  be  achieved  by  soliciting  input  from  experts. 
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